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ABSTRACT 

The Skills Standards Projects have provided further 
emphasis on the need for benchmarking U.S. vocational-technical 
education (VTE) against international competition. Benchmarking is an 
ongoing systematic process designed to identify, as quantitatively as 
possible, those practices that produce world class performance. 
Metrics are those things that are to be measured in a performance 
system. Each must have a measurement process designed to determine 
objectively the status of both the benchma.rker and that which is 
benchmarked on that metric. Types of metrics include the following: 
an exam, number of hours to produce a program completer, operating 
budget per student, placement rate of students in directly related 
jobs, and employer satisfaction with the quality of the completer. 
For benchmarking to have utility in VTE, the metrics chosen must keep 
the process relatively easy and yield results useful for program 
improvement. For a VTE benchmarking model to be of benefit to 
practitioners, it must review elements found in a high quality VTE 
program. The product of the VTE system is an individual prepared to 
be a productive employee. In the design of a system to benchmark VTE 
programs and the products of those programs, there are numerous 
elements to bti considered. They can be identified within two 
categories: those elements of process and those characteristics of 
product. The most compelling reason to benchmark in VTE is the need 
to improve the quality of the product. (YLB) 
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THE VTE BENCHMARKING MODEL 



BENCHMARKING QUALITY PERFORMANCE 

IN 

VOCATIONAL TECHNICAL EDUCATION 
Charles Losh, Ph.D. 

Introduction 

One of the major challenges facing vocational technical education (VTE) is a perception, deserved 
or other v/ise, that VTE programs are not providing "world class'' employees. Another challenge, 
directly linked to the first, is the recommendation that education, and especially vocational 
technical education, benchmark against our international competition. 

Benchmarking is a process widely used by major manufacturers, but one that is not familiar nor 
widely used in education. Recently, the Skills Standards Projects sponsored by the United States 
Departments of Education and Labor have provided further emphasis on the need for 
benchmarking United States VTE at the international level. The opportunity and need to 
benchmark the standards to a world class level were paramount. The problem, however, was a 
lack of information as to how this was to occur. Even more simply, what is benchmarking and 
how could and should the concept be applied to VTE? 

Benchmarking has many definitions but th^^re are common characteristics. In the manufacturing 
models reviewed, there were common themes in all of the processes. Those themes provided the 
basis for The VTE BENCHMARKING MODEL described in this paper. This model has utility 
not only for selected standaros projects, but for entities interested in providing information that 
will enhance the competitiveness of industries and individuals in the United States. 

Benchmarking in Business and Industry 

The International Benchmarking Clearinghouse (1992) describes benchmarking as follows: 

Simply stated, benchmarking is finding and implementing best practices. It is a process of 
comparing your organization and processes to leaders anyu'here in the world and using that 
iniormation to take action that will improve performance, (p. 1) 

Business and industry are heavily involved in benchmarking, almost all FORTUNE 500 companies 
have conducted a benchmarking project, with some viewing benchmarking as a way of life. 



In 1989 the MIT Commission on Industrial Productivity reported that: 

...a characteristic of all the best-practice American firms, large or small, is an emphasis on 
competitive benchmarking. Comparing the performance of their products and work processes 
with those of World Leaders in order to achieve improvement and measure progress... 

Sources reviewed in the development of The VTE Benchmarking Model included articles in 
journals produced by both academics and practitioners, and presentations by practitioners fi*om 
respected and recognized manufacturers. Clearly however, benchmarking processes are still 
evolving, and as Ford (1993) observed: 

At this early point in the development of benchmarking, no single model has emerged to guide 
prospective bencKmarkers. Instead, a variety of competing models exists, encompassing 
anywhere fi-om four to as many as 1 1 steps, (p. 38) 

Although there were differences in terminology and approach, there were common elements in the 
representative literature reviewed. The common elements included concerns that the "metrics" 
(what is to be measured) be clearly identified, and that benchmarking be conceptualized as a 
systematic process. 

Continuous improvement is a theme that provides context for benchmarking, as identified by 
Watson (1992) of Xerox Corporation: 

The purpose of benchmarking is to help a company identify what it needs to improve and how 
it needs to improve by studying the methods used at companies that have been successfijl. 
Benchmarking is an approach to organizational learning that makes comparative, quantitative 
assessments to discover gaps in performance. Orce a gap is identified, the cause of 
performance differences between the companies is analyzed. Companies conduct 
benchmarking studies both to ieam what levels of performance are possible and how that 
performance has been achieved, (p. 38) 

It is clear from Watson's remarks that benchmarking uses specific metrics for quantitative 
comparisons between a benchmarked entity with high quality products and a benchmarkee 
desirous of performance improvement. In benchmarking parlance, these are the "benchmarking 
partners". It is assumed that there is agreement between the parties to participate in the data 
gathering exercise. If agreement and cooperation does not exist, then benchmarking will not 
occur. If there is not agreement and an awareness, then obfijscation of data is the most likely 
result. 

Once the data from the metrics are available, gap analysis is used to determine the need for 
modification of the benchmarkee' s production process. Benchmarking is uhimately an inquiry 
process designed to identify what works and why it works, literally a best-practices approach to 
developing quality. Once a successfijl practice is identified, it becomes a benchmark and serves as 
a reference point for establishing internal goals and objectives for increased performance. 

From these sources it can be seen that benchmarking of international performance, and more 
specifically, benchmarking the quality of world class vocational technical education programs, will 
be an ongoing process. Although there is an obvious need for a snapshot of performance and the 
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characteristics that yielded that level of performance, benchmarking is not simply a snapshot. 
Benchmarking is an ongoing systematic process designed to identify, as quantitatively as possible, 
those practices that produce world class performance. Tliis is tme regardless of whether that 
performance is designed to produce a world class automobile or a world class technician. 

The Metrics 

The use of the term "metrics" in this context is not familiar to vocational technical educators. 
Metrics, as used in benchmarking, does not refer to units of measure in the metric measurement 
system such as meters or grams. Metrics are, most simply stated, those things that are to be 
measured in a performance (competency based) system. Although quantification for comparative 
purposes is a major activity in a benchmarking system, according to Ford (1993), there are 
caveats. 

Although experts caution that benchmarking is not synonymous with measuring, the issue of 
what to measure is clearly one of the thorniest problems confronting would-be benchmarkers. 
Choosing the wrong set of metrics is one of the surest ways to doom a benchmarking study. 
A related problem is making comparisons based on measures that have been calculated 
differently, akin to the age-old "apples and oranges" dilemma, (p. 38) 

For a decision to be made relative to the quality of a system, a common understanding of what is 
important in determining product quality is mandatory. This is critical given that what is 
measured will be what is important. Another way of thinking about this is to ask, "What are 
valued characteristics of the product?" In the case of an automobile, a valued characteristic is the 
number of defects per unit, since defects are ultimately the cause of concern on the part of the 
consumer. In a quality system for producing vehicles, a metric would be defects per unit. Since 
that is a valuable characteristic, the questions become "How does one company produce vehicles 
with such a low defect r^^te?" "What are the production processes that cause this phenomena?" 

Likewise, in vocational technical education, a valued characteristic of a VTE program completer 
is the ability to quickly become a productive employee with limited employer preparation. 
Virtually all new employees have a time frame required for learning the specific routines and 
process of a new employer, but minimization is desired. 

The VTE benchmarker must also choose metrics that impact the quality of the product. These 
metrics include data such as cost per student, number of hours of instruction, and other program 
variables familiar to experienced VTE participants. When choosing the metrics, the most critical 
point is that the benchmarking partners will use the same metrics; no more, no less. No 
comparative analysis can be made on unlike data. This means that the benchmarking partners 
must agree on the metrics, and the customer must also verify the metrics used for product quality 
measures. 

Although each benchmarking process will have its own metrics, figure 1 provides examples of 
potential metrics lor VTE benchmarking. 



. The absolute technical skill level of completers 
. The absolute academic skill level of completers 
. The technical skill gain using pre-post measures 
. The related academic skill gain using pre-post measures 
. The "time on task", both classroom and laboratory 
. The number of hours required to produce a completer 
. The frequency of business industry curriculum input 
. The amount of business industry hard'soft ware contribution 
. The academic qualiiications of the instructor(s) 
. The prior workplace experience of the instructor(s) 
. The state-of-the art status of the instructional hardware 
. The state of-the-art status of the instructional software 
. The appropriateness of the instructional environment 
. The placement rate of students in directly related jobs 
. The placement rate of students in related jobs 
. The placement rate of students in non-related jobs 
. The employer satisfaction with the quality of the completer 
. The operating budget per student 
. The cost per placement 
. The personal characteristics of the students 
. The academic ability level of incoming students 
. The technical ability level of incoming students 
. The opportunities for workplace experience while training 

figure i. 

Although the above list is extensive, the potential benchmarker will be able lO identify metrics that 
fit the proposed benchmarking process. In choosing the metrics to be used in the benchmarking 
process, discipline is critical. Each of the metrics must have a measurement process designed to 
objectively determine the status of both the benchmarker and benchmarkee on that metric. The 
chosen metrics must, when applied, yield like data. Snapshots of status, without comparable data 
on both parties, is of limited or no value. The benchmarker must be able to identify discrepancies 
in performance using quantitative processes to assess comparable processes. 

One of easiest metrics (performance indicators) to identify and use is an exam of the type used by 
the National Institute for Automotive Service Excellence (ASE) to certify automotive technicians. 
The results of this test provide an excellent metric (the absolute knowledge or skill level of a 
completer at a given point in time), assuming the test is administered at the same point in time in 
the individuals experience. Other metrics that might be chosen include the number of hours to 
produce a completer, the operating budget per s'.udent, the placement rate of students in directly 
related jobs, employer satisfaction with the quality of the completer, and others from figure 1 or 
determined by the benchmarkers. For benchmarking to have utility in VTE, the metrics chosen 
must keep the process relatively easy and yield results that are usefiil for program improvement. 
In the ASE example, suggested performance metrics would be high ASE scores, a high placement 
rate as an automotive technician, and employer satisfaction with the employee's performance. 



Discipline will be required to agree on common metrics that can be applied to all VTE programs. 
Without the discipline of common metrics, program improvement will continue to be ad hoc. The 
best practices will continue to occur on a limited basis, which does not move the VTE enterprise 
forward. 

Benchmarking Vocational Technical Education 

For a VTE Benchmarking Model to be of benefit to practitioners, it must review the elements 
found in a quality vocational technical education program. Quality is a subjective term that is 
diflBcult to conceptualize, and is nearly always in the eye of the beholder. Recently, however, 
quality has been defined for business and industry in ISO 8402 (the International Standard for 
term definitions for the ISO 9000 series of standards) (Rothery 1993): 

...the totality of features and characteristics of a product or service that bear on its ability to 
satisfy stated or implied needs. 

The VTE Benchmarking Model will use the ISO definition of quality. 

A quality product of the VTE system is an individual prepared to be a productive employee of an 
organization. Productive in all senses of the word, 

including not only those technical skills required to interact with the technical environment, but 
also those skills required to positively interact with the human environment. 

In the design of a system to benchmark VTE programs and thereby, the products of those 
programs, there are numerous elements to be considered. Those elements can be identified within 
two categories: those elements of PROCESS, and those characteristics of PRODUCT. Looking 
at either of these isolated from the other will not yield adequate information to determine those 
system characteristics that resulted in a quality product. 

Simply stated, the purpose of benchmarking is to identify and model quality performance. The 
VTE Benchmarking Model is shown in figure 2. 

The VTE Benchmarking Model 

1 . Identify the metrics to be used. 

2. Use the metrics to assess a quality VTE program. 

3. Use the metrics to assess the VTE program to be improved. 

4. Identify performance levels achieved by each partner on all metrics. 

5. Idep*'fy how high performance was achieved by the quality program. 

6. Identify what needs to be modified to improve the inferior performer. 

7. Identify how the performance on selected metrics can be improved. 

8. Apply the improvement tactics. 

9. Re-assess using the metrics to determine performance improvement. 

figure 2. 
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Using The VTE Benchmarking Model, a benchmarking instrument can be developed to meet the 
needs of the specific VTE program to be benchmarked. A critical point, however, is that for 
benchmarking to be truly effective step eight must include activities that will cause change. 
Numerous change models exist, but establishing goals and providing mechanisms to enable 
change are critical. Without action on the existing system using data from the benchmarking 
exercise, quality will not improve. Simply conducting selected steps of the process will not yield 
changes in qujility. Commitment to improvement by all stakeholders must be in place for results 
to accrue from the benchmarking process. 

The data from the process metrics will show the possible level of performance on each metric. 
This data will provide indicators of the practices of the quality program that can ultimately be 
adopted by the program to be improved. There will be gaps or discrepancies in performance 
between the programs, and analysis of the discrepancies will identify the cause of performance 
differences that determine areas needing improvements. 

For example, the number of hours (time on task) may be less in the lower performing program. 
This would target an area for change, although there is certainly no guarantee that more hours 
will produce a higher quality product. The benchmarking study would include what happens 
during those hours. Seat time is generally not the answer to higher performance, but the quality 
of those hours will impact performance. 

Most simply stated, performance improvement will occur through adoption of the processes of 
the successful enterprise, replicating what works in the quality 

program. Review of the data from the metres will identify those areas that differ between the 
benchmarking partners. Once identified, those areas or processes need to be modified to more 
closely resemble those of the more effective program. Policy level commitment must be gained 
for the changes, since in many instances the changes will be significant, and differ radically froni 
present practice. Improvement may or may not come easily. In some instances, relatively simple 
modifications of process will yield radical improvements in product. 

On some metrics, comparison of the data will show very small differences in performance. Where 
there is small data variance, there is no need to develop objectives for process changes. 

Recalibration of the metrics to reflect experience in the benchmarking process will be required as 
part of the improvement process. It is unlikely that the initial benchmarking exercise will yield the 
desired improvement without repeated cycles through the process. One must ask the obvious 
question, "Is the program performance improving on the metric(s), and if not, why not?". Is it a 
commitment problem or a measurement problem? 

H//?y Benchmark? 

The most compelling reason to benchmark in vocational technical education is the need to 
improve the quality of our prodi If elements of the production system can be improved with a 
concomitant improvement in product quality, then those elements must be identified and 



improved. Follow through on the identification of differences in performance on the metrics will 
yield program improvement. 

Benchmarking is a tool for vocational technical educators continuing to strive for excellence. 
Determination of the elements that yield quality programs will raise the excellence of all programs. 
The VTE Benchmarking Model is a tool to guide the VTE benchmarking process, and thereby 
contribute to the goal of quality VTE products and programs. Benchmarking, carried out on a 
continuous basis, will improve the enterprise. 
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